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Evaluation System and Method 

Field of the invention 

The present invention relates to a computer based system and method for carrying out an 
evaluation, and in particular for performance measurement and evaluation within a group based 
5 organisational structure. 

Background of the invention 

Performance within an organisation or between organisations is often measured by 
means of performance reviews or the like. A less than optimal structure often present in 
performance reviews can lead to a number of undesirable behaviours amongst an organisation's 

10 staff. For example, in organisations where an individual's immediate superior has a large 
influence on a performance review outcome, staff behaviour tends to become of an ingratiating 
nature towards their immediate superior with little regard for peers and team behaviour. Also, 
performance evaluation can often be a tedious process and can consequently be implemented 
very poorly. Further, if diverse inputs are solicited from many different sources to form the 

15 basis of a performance evaluation then they may be poorly integrated together. Often, the 
individuals or structures being reviewed may not be well aware of an organisation's values and 
the performance evaluation may not properly measure the staff adherence to such values. 

Summary of the invention 

In accordance with a first aspect of the invention there is provided method of providing 
20 an evaluation on a computer system, the method comprising the steps of: (a) selecting a 
plurality of evaluation respondents; (b) selecting a plurality of groups into which the evaluation 
respondents are divided; (c) creating at least one interactive evaluation session for the groups of 
respondents, the evaluation session comprising a series of questions; (d) storing the 
respondents, groups and evaluation session on the computer system, (e) providing each of said 
25 respondents with the opportunity to interactively answer the questions in the evaluation session, 
(f) storing the answers on the computer system and (g) generating a series of reports based on 
the answers, wherein different types of reports can be generated by selecting different groups as 
a basis of comparison. 
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Preferably the computer system includes a relational database, and at least some of the 
evaluation respondents are divided into more than one group, with the relational database being 
set up to enable the different groups to be selected as a basis of comparison. 

Conveniently, the groups include primary groups and non-primary or secondary groups, 
5 with the groups being selected such that a respondent may belong to only one primary group 
and to one or more secondary groups. 

The evaluation session conveniently comprises a number of substantially independent 
surveys each having a series of questions for answer by a respondent. 

Typically, the method is operated in an internet browser/internet server environment. 

10 Preferably, the environment includes a host server having an associated relational 

database, at least one facilitator browser computer and a plurality of respondent computers, with 
the facilitator being involved in the initial selecting and creating steps. 

Advantageously, the series of questions are arranged to require a response whereby 
respondents rank subjects or objects, for enabling the generation of ranking reports. 

15 Step (e) typically includes the steps of launching the evaluation process by including a 

universal resource locator (URL) of the address of a corresponding evaluation or survey to be 
carried out by the respondent. 

The invention extends to a computer system for producing an interactive evaluation 
comprising respondent and group organisation means for inputting respondent and group details 

20 into said computer system of the respondents and groups participating in the evaluation, 
evaluation creation means for creating, for each respondent in a group of respondents, an 
interactive evaluation including a series of questions, notification means for notifying the 
respondents of said group of the availability of the corresponding interactive evaluations, 
evaluation execution means for enabling respondents to interact with a corresponding 

25 evaluation, storage means for storing the evaluation result in respect of each respondent, and 
reporting means for collating and reporting said evaluation results. 

The system preferably includes monitoring means for monitoring the current status of 
the interactive evaluation. 
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Typically, said monitoring means includes a second notification means for sending a 
reminder to respondents who have not completed their evaluation. 

The system conveniently includes registration means for initially registering an 
organisation's details within said computer system. 

5 Preferably, the respondent and group organisation means comprises a set up graphical 

user interface (GUT) for enabling a facilitator to enter and arrange names and into primary and 
secondary groups. 

The evaluation creating means advantageously comprises at least one evaluation GUI 
including a survey template option providing different survey types to choose from, and a 
1 0 question bank including questions associated with at least some of the survey types. 

The survey type is preferably chosen from a group including a self review, a colleague 
review, an opinion poll, and confidential and non-confidential reviews. 

The evaluation execution means typically includes a plurality of GUI's carrying 
respondent-specific survey questions. 

15 The computer system advantageously includes closure means for enabling a facilitator to 

close an evaluation by force closure or deletion. 

The reporting means may include select means for selecting a reports option available to 
an identified subject, the reports option being respondent-and group-based. 

The reports option preferably includes a ranking option, whereby the identified subject 
20 can select which subject-accessible groups or respondents can be ranked by which other groups 
or respondents. 

The reporting means may be arranged to process textual answers, and includes an edit 
facility to enable a user to moderate or bowdlerise the textual answers prior to saving them. 

The invention extends to a method of providing an evaluation on a computer system, the 
25 method comprising the steps of providing an interface for facilitating the initiation of an 
interactive evaluation session, providing a respondent and group entry interface for enabling the 
entry of groups of evaluation respondents, providing an evaluation creation interface for 
enabling the compilation of at least one targeted survey for the respondents, providing an 
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evaluation launching interface for launching the survey to the respondents, providing each of 
the respondents with the opportunity of interactively answering questions in the survey, 
receiving and storing on the computer system details of the evaluation session, respondents, 
groups, surveys and answers, and providing a report selection interface for enabling the 
5 selection of a series of reports on the survey. 

Preferably, the group and respondent details on the evaluation session are stored on a 
relational database in such a way that allows for the selection of different combinations of 
groups at the reporting stage. 

A method may further include the step of allowing the entry of different groups of 
10 evaluation respondents as primary groups and secondary groups, with respondents being limited 
to membership of only one primary group, but to more than one secondary group. 

Conveniently, the group and respondent details are stored in tables which include a 
group table containing details of the groups being evaluated, and a membership table including 
details of the primary group membership in respect of a given evaluation, and all secondary 
1 5 groups in respect of the evaluation. 

A further aspect of the invention provides a computer system for enabling a computer- 
based evaluation, the system comprising initiation means for generating an interface for 
facilitating the initiation of the interactive evaluation session, respondent and group entry 
interface generation means for enabling the entry of groups of evaluation respondents, 

20 evaluation creation interface generation means for enabling the compilation of at least one 
targeted survey for the respondents, evaluation launching generation interface means for 
launching the survey to the respondents, enabling means for providing each of the respondents 
with the opportunity of interactively answering questions in the survey, storage means for 
storing on the computer system details of the evaluation session, respondents, groups, surveys 

25 and answers, and report selection interface means for enabling the selection of a series of 
reports on the survey. 

The computer system may include server means incorporating storage means in the form 
of a relational database. 
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According to a still further aspect of the invention there is provided a computer program 
product comprising a computer readable medium having its own program code means when 
said program is loaded, to make the computer execute procedures in accordance with the 
method as set out above. 

The invention extends to a server hosting a web page database wherein the graphical 
user interfaces described above are in the form of web pages, and in combination define a web 
page database. 

These and further features of the invention will be made apparent from the description of 
a preferred embodiment thereof given below by way of example. In the description references 
made to the accompanying drawings but the specific features shown in the drawings should not 
be construed as limiting on the invention. 

Brief description of the drawings 

Preferred and other embodiments of the present invention will now be described with 
reference to the accompanying drawings in which: 

Figure 1 illustrates schematically an Internet type environment within which the 
invention operates, including the architecture of the evaluation system; 

Figure 2 is a flow chart of the registration process of the invention; 

Figure 3 is a flow chart of the organise process of the invention; 

Figure 4 is a flow chart of the create process of the invention; 

Figure 5 is a flow chart of the edit process of the invention; 

Figure 6 is a flow chart of the open or launch process of the invention; 

Figure 7 is a flow chart of the manage or progress process of the invention; 

Figure 8 is a flow chart of the close process of the invention; 

Figure 9 is a flow chart of a report process of the invention; 

Figure 10 is a flow chart of the evaluate or respond process of the invention; 

Figure 11 shows a screen shot of a main manage evaluations page; 
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Figure 12 shows a screen shot of part of a subject details set up page; 



10 



15 



20 



Figure 12 


shows 


Figure 13 


shows 


Figure 14 


shows 


Figure 15 


shows 


Figure 16 


shows 


Figure 17 


shows 


Figure 18 


shows 


Figure 19 


shows 


Figure 20 


shows 


Figure 21 


shows 


Figure 22 


shows 


Figure 23 


shows 


Figure 24 


shows 


Figure 25 


shows 


Figure 26 


shows 


Figure 27 


shows 


Figure 28 


shows 


Figure 29 


shows 


Figure 30 


shows 


Figure 31 


shows 


Figure 32 


shows 


Figure 33 


shows 


Figure 34 


shows 


Figure 35 


shows 



Ovl vvlJ. 


shot 


of a 


Ovl wll 


shot 


of 3. 


screen 


shot 


of a 


screen 


shot 


of a 


screen 

Ovl JUL 


shot 


of a 


screen 

Ovl vvll 


shot 


of a 


screen 


shot 


of a 


screen 


shot 


of a 


screen 


shot 


of a 


screen 


shot 


of a 


screen 


shot 


of a 


screen 


shot 


ofa 


screen 


shot 


of a 


screen 


shot 


ofa 


screen 


shot 


ofa 


screen 


shot 


ofa 


screen 


shot 


ofa 


screen 


shot 


ofa 



SUBSTITUTE SHEET (RULE 26) 



WO 02/097683 



PCT/AU02/00722 



7 

Figure 36 shows a screen shot of a report listing page; 

Figure 37 shows a screen shot of typical confidential report; 

Figure 38 shows a screen shot a typical non-confidential report; 

Figure 39 shows a table representative of a ranking report; and 

5 Figures 40A and 40B show a schematic diagram of part of the relational database of the 

evaluation system of the invention. 

Description of preferred and other embodiments 

In the preferred embodiment, there is provided an interactive performance evaluation 
system operational over an Internet browser type environment. Turning initially to Figure 1, 

10 there is illustrated schematically a standard browser environment wherein a facilitating client 
computer and browser 1 interacts over the Internet 2 with a server application 3 running on 
another host computer. Other client computers which are linked both to the facilitating client 
computer 1 and the server application 3 include respondent computers and report receiver 
computers 4 and 5 respectively. The server application can operate in accordance with standard 

15 common gateway interface (CGI) techniques, and incorporates a relational database 6 which is 
populated by the facilitator and respondent computers. 

The application is provided for serving web pages over the Internet so as to allow an 
organisation to perform comprehensive evaluations of their staff performance, team 
performance, organisation performance and staff opinion as required or on as many issues as 
20 required. Further, the application can be used for evaluating anything by anyone in as much 
non-human objects (for example films or detergents) can be the subjects of a survey and anyone 
at all can be a respondent of a survey. Specifically the functions/interfaces provided by web 
pages served by the server 3 can include the following (described in more detail below): 

1. Register - Organisations to initially register with the system and designate an 
25 administrator or facilitator. Facilitators act on behalf of the organisation to: 

2. Organise — For the organisation, review entered details, upload corporate 
information such as a logo, enter the staff (and other people or objects that they might require to 
be respondents or subjects of surveys), designate supervisors, assign other 
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admiiiistrators/facililators, enter entities or groups, relate groups to each other, relate groups to 
staff, and designate which staff can see which results for which groups; 

3. Create - Create evaluations either from a template, a previous evaluation or from 
a designated list of module or survey types selected by the facilitator; 

5 4. Edit - Edits evaluation details, survey lists, and for each evaluation: designates 

the evaluators or respondents, designates who or what are to be evaluated, assigns questions and 
assigns multiple choice answers to questions; 

5. Launch - Launch or Open an evaluation and send out invitations to all evaluators 
or respondents; 

10 6. Progress - Generate reports on the status of an evaluation and send out reminders 

to delinquents; re-open individual sessions on request; 

7. Close - Close an evaluation and make reports on the results of the evaluation 
available; 

8. Report - Generate reports for organisation-wide surveys and for staff, if 
15 necessary; 

9. Respond - Respondents or Evaluators to complete an evaluation or survey when 
invited; they are presented with only the surveys for which they have been designated as an 
evaluator; 

Register: 

20 Figure 2 illustrates an example flow chart 10 of the steps involved in the registration 

process. This flow chart shows how a representative or facilitator of an organisation can enter a 
web site address 11 to provide for initial registration. The representative is asked to enter 
minimal organisational information 12 including naming a facilitator and their email address. 
An initial password is then sent 13 to this facilitator; to at least allow the email address to be 

25 authenticated. The facilitator is also sent an email containing a special URL. On clicking this 
URL 14, the facilitator is requested to enter the password contained in the email; once this is 
done the facilitator enters his or her own secret password 15 (twice to preventing typing errors). • 
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After passing through a "Welcome" page 16, the facilitator is then logged in and can proceed to 
the next step of Organise, or another facilitator function 17 of the facilitator. 

Organise: 

Figure 3 illustrates an example flow chart of the steps 20 involved in creating an 
5 organisation profile within the system. The flow chart shows four different selectable functions 
21 to 24 that the facilitator has at their disposal to set up the organisation ready for evaluations 
to be created. 

The first function that the facilitator is likely to use is Edit Organisation Details 22. This 
allows them to review existing details, add country (mandatory for GST purposes), set system 
1 0 language preference and upload the organisation logo. 

The second function that can be used is to upload a staff file 23. This can be done by 
first producing a spreadsheet containing the fields as predefined by the system. This can be done 
by a variety of methods as required and may be created by extraction of information from 
existing databases run by the organisation. If no such data exists, the facilitator can enter the 
15 details with the Edit Staff function. One key field in the upload is the primary group. A 
primary group is defined as the one group (often department or group) to which a person 
belongs. As the upload process encounters new groups, they are created in the system database. 

The facilitator is able to alter entries using the Edit Groups functionality 24. This allows 
the facilitator to add secondary groups and relate staff to them. It also allows the facilitator to 
20 relate groups to each other and designate one to be the son of another which is used in report 
generation at a later date. 

The Edit Staff function 21 allows the facilitator to review what has been uploaded, add 
new staff and relate staff to secondary groups. At this time the facilitator can also designate 
which staff can view reports on groups and which staff can be co-facilitators. 

25 Create: 

Figure 4 illustrates the exemplary steps 30 involved in a facilitator creating an evaluation 
initially. After entering 31 the name, description and target dates for launch and close, the 
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facilitator can decide 32 whether to specify what surveys constitute the evaluation 33 or derive 
this previous evaluation from a list 34. 

Edit: 

Figure 5 illustrates the exemplary steps 40 involved in editing an evaluation. A created 
5 evaluation is chosen 41, and surveys can then be named, renamed, added or removed 42. For 
each survey 43, the list of respondents, report permissions, questions and multiple-choice 
answers are specified and added 44. If the survey requires a set of subjects (people or objects) 
to be evaluated, these are specified here also. 

Launch: 

10 Figure 6 illustrates a flow chart of how a facilitator opens or launches an evaluation 50. 

Once the facilitator is satisfied with the creation and editing of an evaluation and the start date 
has arrived, the facilitator instructs the evaluation system to launch it 51. The evaluation system 
needs to run some checks 52 to ensure that the staff list is valid, the group relationships are 
consistent (no loops) and that all details of the evaluation are consistent. When this is so, the 

15 evaluation system asks the facilitator to provide the text of the opening email 53 to be sent to all 
respondents before formally opening the evaluation, confirming the list of respondents 54 and 
sending the email to all respondents. Prior to this event taking place, the evaluation system asks 
the facilitator 55 which of the new questions that the facilitator has created can be placed in the 
question bank. This question bank is open to everybody and enhances the value of the service. 

20 If the facilitator agrees 56 to donate them, the facilitator may optionally be asked if they want 
the organisation name to be placed by the example. This is advantageous for two reasons: first, 
facilitators of the organisation can quickly recognise and reuse their questions and secondly, the 
organisation gets advertising and value from their donation to the bank. 

Within each email sent 57 is a unique URL that describes the facilitator's name, 
25 organisation and evaluation so that the log-in will only require the entry of a password. 

Progress: 

Figure 7 illustrates a flow chart 60 of the steps in how a facilitator manages the progress 
of an evaluation. A 'View Progress' selection on an evaluation in progress is made 61. A 
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report 62 is then generated showing which respondents have not started, started but not finished 
and finished their part in the evaluation. The facilitator is given the opportunity of sending 
reminder emails 64 to either those who have not even started or those that have started but not 
finished. The facilitator is also given the opportunity to "unfinish" a session 63. This may be 
5 necessary because a respondent may have prematurely finished a session and might want to go 
back and add more feedback. For those respondents who have not even started 65 or who are 
still busy 66 with evaluations, reminder emails 67 are sent. 

Close: 

Figure 8 illustrates a flow chart 70 of the steps involved in a facilitator closing an 
10 evaluation. It may be the case, despite a series of reminder emails, that when a facilitator comes 
to close an evaluation, there will be unfinished evaluation sessions. The facilitator selects the 
close option 71, and is then given a summary of the unfinished sessions 72 and is given the 
choice on each of them to either automatically finish the sessions or delete them together with 
all the associated answers and rankings. On closure 74, all people who can review reports on 
15 the evaluation are subsequently notified by email 75 giving them another unique URL to aid 
quick log-in; these are the supervisors of staff affected and those people designated as being 
able to review reports on specific groups. 

Report: 

Figure 9 illustrates a flow chart 80 of the steps involved in report production. The flow 
20 chart 80 shows how a facilitator can produce reports after an evaluation has been closed 81. 
These reports can be survey specific. A list of available reports is produced 82. The list will 
vary depending on the report permissions set up during the create phase. If the person is the 
nominated reviewer for a report then a link will appear for the option to review the report. 
Every report that has textual answers in it must be reviewed. The reviewer of each report is 
25 specified by the facilitator during the create phase. The user simply selects the required report 
83, enters any required parameters, receives the report on their screen 84 and prints it 85 via the 
browser functionality if required. 

Evaluate: 
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Figure 10 illustrates a flow chart 110 of the steps* involved in the main process of 
respondents performing an evaluation. Although the steps form only a small part of the system, 
such steps constitute the main point of interaction of most users of the system. After logging in 
via the special URL sent from the open evaluation process 1 1 1, an evaluator or respondent will 
5 be presented with a welcome page and a list of modules or surveys for which he or she is 
required to be an evaluator or respondent 112. This will vary from respondent to respondent. 
On pressing NEXT from the welcome page 1 13, the respondent answers questions, ranks people 
or objects, assesses performance against objectives 114, 115 or whatever is required by the 
survey or page before pressing NEXT to select a different page from the navigation bar 115. 

10 This occurs after the system has saved all answers and ranks 116. Once all pages have been 
processed, the respondent is taken into the Finish survey 117 where he or she is asked questions 
on the evaluation or survey itself (specified, if required, by the evaluation or survey creator). In 
this survey or page a general check is made to ensure that that the respondent has performed all 
mandatory actions 118 before allowing them to finish and commit their results to the database 

15 119 and form part of the evaluation data. 

The various processes described with reference to the flow charts will now be discussed 
in more detail with reference to a series of screen shots or portions thereof embodying 
graphical user interfaces (GUI's) 

After the final step of the registration procedure, in which the welcome page is displayed 
20 at 16, the client or facilitator is informed that a certain level of Java needs to be installed as a 
plug-in to the browser. If required, a Java download and install will be automatically initiated, 
this being part of a standard process provided by Sun Microsystems. 

The initial page providing entry into the organised or managed process of the invention 
is shown at 120 in Figure 11. A menu bar 121 allows the user to set up a staff7customer (ie 
25 subject) list or to create an evaluation. All draft evaluations, evaluations in progress and closed 
evaluations are summarised on this page at 122, 123, and 124. Once the facilitator has added 
staff or other subjects and created evaluations, this page will have fewer instructions and will 
show all the subject's evaluations in their relevant "states". 
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Referring now to Figure 12, the first task that the facilitator needs to perform is to set up 
a people database by selecting a set up staff/customer list. The so-called Staff/Customer details 
page 125 of Figure 12 is presented, with, the subject details 126 at this stage including only 
those of the facilitator. The facilitator then needs to enter more staff or other subjects either by 
5 typing them into the matrix provided or by preparing and loading a spreadsheet by selecting the 
option under "Tools". The screen shot of Figure 13 shows a spreadsheet loading page 127 for 
enabling bulk entry of staff/customer or other details. This is achieved by downloading and 
saving the sample Excel spreadsheet, populating it with staff details, clicking on the browse 
button to select the spreadsheet from the system, and clicking on the upload staff/customer 
1 0 spreadsheet button 128. 

Referring now to Figure 14, once the subjects have been loaded, they are organised into 
groups by selecting the organisation or "arrange names into groups" tab 130 and dragging and 
dropping groups and names into primary groups 131 and secondary groups 132. Each person 
can only be a member of one primary group 132, but can be a member of any number of 
15 secondary groups 132. This forms the basis for powerful reporting capacity, as it ultimately 
allows the feedback of any one group on another group to be reported. 

Once the primary and secondary groups of the organisation have been defined and 
populated with subjects, the facilitator may choose to edit the organisation details or even to 
upload the organisation logo, which could then be displayed on each screen shot. 

20 The creation process will now be described with reference to Figure 15, which shows a 

screen shot of a create evaluation page 133. The name of the new evaluation is entered into text 
box 134, and the style of evaluation is then chosen either by clicking on the fic blank" link 135 or 
on the name of a previous evaluation. The blank evaluation is generally created in the first 
instance, and Figure 16 shows a screen shot of a blank evaluation page 136 with the evaluation 

25 name, description, target launch date and target end date having to be filled in respective text 
boxes 137, 138, 139 and 140. An add survey link 141 is also provided for enabling the 
facilitator to add different types of surveys to the evaluation- in this case broadly categorised 
into confidential, non-confidential and colleague review. 
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Once a survey is added, the focus turns to the actual survey. An alternative way of 
adding a survey is shown in Figure 17. A survey template window 142 provides various survey 
options, some of which are set out in the window 142. The selected survey is simply dragged 
and dropped into the evaluation, and appears in a tree structure 143 in the left panel of Figure 
5 18, which shows part of a sample survey or evaluation 144. Questions can be modified and 
inserted either by entering new ones or dragging existing ones in from a question bank window 
145 including categorised questions obtained from the tools menu. If the survey is based on 
people as subjects, the facilitator also needs to add the names and other relevant details of the 
subjects who need to be assessed by the questions, by clicking on a subjects tab. 

10 Figure 19 shows a colleague review screen shot 146 in more detail. A colleague review 

has a number of selectable features. First, the facilitator can choose whether the questions are 
asked all at once in respect of each subject, as is shown at 147. Alternatively, or in addition, 
respondents can answer one question at a time on a list of subjects 148. A ranking option 149 is 
also provided, enabling the facilitator to choose that the respondents rank the subject in order of 

15 value so that a unique value ranking can be created. The colleague review also allows 
respondents to choose which people to answer questions on. This constitutes a subset of the 
subject selected from the staff/customer list. 

Once each survey has had its details completed, the various questions have been added 
and the respondent's (and subject, if required) have been added, the facilitator then needs to 
20 specify who needs to receive the reports. 

Figure 20 shows a page incorporating a survey report by group window 151 providing 
the facilitator with option blocks 152 for enabling the facilitator to specify exactly who will 
receive the reports. Depending on the type of report, different report permissions are provided. 
Once the report selection process has been completed, the facilitator can check that the 
25 evaluation is ready for launching. A check window (not shown) may be provided for ensuring 
that further information is incorporated. For example, respondents may be added to the self 
assessment survey, and report permissions may be required for the "survey report by person" in 
the self assessment survey. Part of a survey launch page is shown at 1 53 in Figure 21 . 
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Prior to launching the evaluation, the facilitator can select at 154 if any questions 
authored can be used in the question bank, and the subject and text of the email inviting people 
to participate in the evaluation are incorporated in respect of the text blocks 155 and 156.The 
confirmation screen 160 of Figure 22 is displayed on launching the evaluation, and includes full 
5 details of the respondents who have been sent an email inviting them to participate. 

As is clear from Figure 23, each respondent automatically receives an email 161 asking 
the respondent to select a link and set their password. Figure 24 provides a first employee 
opinion survey page 162, which is the first page of the first survey. A left-hand panel 163 
describes the contents of the evaluation, and includes link to colleague review, person, value, 
10 ranking and self-assessment. The status of each survey is depicted by a traffic light icon 164. 

A first colleague review page is shown at 165 in Figure 25, with the left-hand panel now 
including a list of colleagues to be reviewed or evaluated 166. 

Figure 26 shows at 167 a second colleague review page incorporating a comparative list 
of colleagues dealt with on an issue by issue basis. 

15 In Figure 27, a third colleague review page 168 provides a series of text boxes 169 in 

which questions on various colleagues can be answered on an issue by issue basis. Figure 28 
shows a fourth colleague review page 170 including a series of score boxes 171 for enabling a 
list of selected colleagues to be scored out of 10. 

In Figure 29, a fifth colleague review page 172 includes tick in the box questions about 
20 colleagues which can be dealt with on an issue by issue basis. 

Figure 30 shows a self assessment page 173 providing a series of text boxes 174 for text- 
based responses. In Figure 31, a colleague review ranking page 175 allows respondents to rank 
one another in value order, providing a series of text boxes 176 of differing levels, and in this 
case ranked from high to low. This constitutes a significant feature of the invention, in that by 
25 collecting individual ranks of subjects by respondents, the system is later able to produce a 
ranking of any group or set of groups as ranked by any other group or set of groups. The 
respondent ultimately proceeds to a Finish page where the submission will be submitted. This 
is analogous to signing a form. Once this has been done, the respondent cannot return to answer 
further questions without requesting for the session to be "unfinished" by the facilitator. 
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Once the evaluation is launched, it acquires an "in progress" status. In the first review 
page 177 of Figure 32, the facilitator is able to view the progress of the evaluation by selecting 
the relevant option from the relevant drop down list 178. hi the second review page 179 of 
Figure 33, a full survey summary is provided providing the statuses of all the respondents. The 
5 facilitator is then able to review the progress of the response and can choose to delete or to 
"unfinish" sessions 179A or to send out reminders to respondents who not have not yet finished. 

Once the facilitator decides to close the evaluation, this can be done provided there are 
no started but not finished sessions. Part of a closed evaluation page is indicated at 180 in 
Figure 34. The facilitator then authors an email that is sent to all those people who have been 
1 0 designated to receive reports. 

After closing, a confirmation report showing which people have been sent emails about 
reports for the evaluation is then provided, and an email informing the respondents that the 
evaluation is closed is then sent to all respondents, who are also invited to review reports. 

On returning to the manage evaluations page from the closed evaluation page, the 
15 facilitator can then see that the evaluation has been closed and can be selected from drop-down 
menu 178 for report viewing, as is shown in Figure 35. The user is then presented with a list of 
the reports that he or she is entitled to view for the evaluation in one of two ways. If the user is 
a facilitator, the above drop-down menu selection may be made. Alternatively, the user may 
receive an email inviting him or her to review report, after which the link in the email is clicked 
20 on and the user then is able to sign in. As is clear from the report listing page 181 of Figure 36, 
the user can then view a selected report or can review the textual answers of a report if such 
user is a designated viewer for the report. If the user selects to review the textual answers of a 
report, the user is then presented with a page of all of the answers entered, and is given the 
ability to edit it to summarise them. This is considered necessary in the case of a confidential 
25 report, on the basis that inflammatory or counter-productive comments may possibly have been 
made, with the result that the company will need to protect itself and its employees. Once the 
answers are reviewed, the resulting text is made available in the report. If the report is reviewed 
by someone before the text answers have been reviewed, the answers are not available, and the 
report viewer is advised which review they are awaiting. 
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On selection of a confidential report, an authorised user is presented with it immediately, 
it is shown from part of a confidential report page 182 in Figure 37. This page shows the 
beginning of a confidential report, which means that individual answers are protected. Figure 38 
shows a page 183 of part of a non-confidential report, including individual answers underneath 
5 each group entry. 

A summary of all of the answers given by group and sub-group of both primary and 
secondary groups is presented. This is an extremely powerful analytical tool, and it can be seen 
why secondary groups allow the facilitator to get different views on different subjects by 
allowing different cross-sections of the organisation to be formed. It is possible to generate 
10 many different types of reports once the database structure is in place. 

The way in which the database is assigned allows a complex organisational structure to 
be captured through the facilitator interface, and for the structure subsequently to be analysed on 
numerous different group-based levels. 

Ranking reports represent a more complex version of the report. A user is given rights 
15 to view ranking reports of one or a set of different groups. When the ranking report option is 
selected, the user is presented with a first list of the groups for which they are allowed to view 
the results (which could be only one). These are known as the subject groups. The second list 
is of all the groups that have members who have ranked or voted on any of the members of the 
subject groups. These are known as respondent groups. The user is enabled to get a ranking 
20 report on a combination of subject groups as voted on by any combination of respondent 
groups. A typical ranking report is shown at 184 in Figure 39, and includes name and group 
details, together with a relative score column 185 and details of the number of respondents 186 
in each score, hi order to exclude inconsistent or freak results, the system is typically set up not 
to include subjects received fewer than a critical number- say five- ranks or votes. 

25 It can be seen that any subject can be given a ranking position within any group for 

which they are a member based on the rankings (votes cast by any combination of groups). 
These positions can be reported in an individual's Person Reports. 
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The most common report is to view the whole organisation as ranked by the whole 
organisation. It is also possible to drill down and to view the results of any subset of the 
organisation on the votes cast by any other subset (or the same subset). 

The system will also allow "objects" to be ranked by Respondents. Objects are very 
5 similar to subjects; they can be arranged in group structures and reported on in the same way. 
An object can be anything that is comparable with another set of objects. Typical examples are: 
films, shows, detergents, sports teams, political parties, and individual performances. This 
allows Respondents to be requested to answer questions on and rank not only people but also 
any set of objects that they want compared. 

10 Films, for instance, would be categorised in some way, say: drama, thriller, comedy, 

action. They may also be categorised using secondary groups by say; censorship level, actors, 
or indeed any other factor. The Respondents can then be grouped by area with subgroups (e.g. 
NSW, Sydney, Mosman) and then be categorised further by cross-section using secondary 
groups such as sex, age bracket and the like. The Respondents are then asked questions about 

15 all the films and asked to rank them. Reports can then be generated on any subset of the films 
as assessed by any subset of the Respondents. 

Subjects and objects can accordingly be ranked with the ability to group hierarchically in 
both primary groups and secondary groups, with both being combined in reports of any kind. 

Referring now to Figures 40 A and 40B, part of a relational database 190 is shown 
20 illustrating the key tables that implement the system of the invention, together with the 
relationship between the tables. Each table consists of several fields which are labelled with the 
table boundaries. Key icons indicate the primary key and the lines show the relationship 
between the various tables. The tables include a person table 191, an answer table 192, a survey 
question (called ModuleQuestion) table 193, a session table 194, a ranking table 195, an 
25 "evaluation group" (called EvaluationEntity) table 196 containing the various groups being 
evaluated or surveyed, and an "evaluation of group member" (called EvaluationEntityMember) 
table 197 including a membership for a given group in respect of the given evaluation. 

An evaluation of all group members (called EvaluationEntityMemberAll) table 198 
contains a membership for a given group and all its sub-groups in respect of the given 
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evaluation or survey. In Figure 40B, an "answer detail" table 199 provides details of all the 
answer choices in a multiple choice question, whilst the "answer" table 192 contains the actual 
answers which refer to either answer choice or a textual answer or contains a number. A 
question table 200 includes various question types, a survey table 201 includes the various 
5 different surveys or modules and an evaluation table 202 includes details of the evaluation itself 
such as name, description and start and end dates. 

The relationship between the various tables will be apparent to one of normal skill in the 
art on the basis of what is included in Figures 40A and 40B. Foreign keys in tables relate to 
primary keys in other tables, and the relationships can be derived from the names. For example, 

10 tblAnswerl92 has three foreign keys relating to primary keys of other tables: anPEID to peBD in 
tblPerson 191, anSEID to selD in tblSession 194 and anMQID to mqlD in tblModuleQuestion 
193. The answer table actually has two other foreign key links: aTEID relates to the text table 
where textual answers are held and a USID relates to the user session table which holds details 
of the users session. It can be seen that this design allows a facilitator to collect and analyse the 

15 opinions (both answers to questions and rankings) of any set of groups on any other set of 
groups. 

It will be understood that the invention disclosed and defined herein extends to all 
alternative combinations of two or more of the individual features mentioned or evident from 
the text or drawings. All of these different combinations constitute various alternative aspects of 
20 the invention. 

The foregoing describes embodiments of the present invention and modifications, 
obvious to those skilled in the art can be made thereto, without departing from the scope of the 
present invention. 
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Claims 

1. A method of providing an evaluation on a computer system, the method 
comprising the steps of: 

(a) selecting a plurality of evaluation respondents; 

5 (b) selecting a plurality of groups into which the evaluation respondents are 

divided; 

(c) creating at least one interactive evaluation session for the groups of 
respondents, the evaluation session comprising a series of questions; 

(d) storing the respondents, groups and evaluation session on the computer 

10 system; 

(e) providing each of said respondents with the opportunity to interactively 
answer the questions in the evaluation session; 

(f) storing the answers on the computer system; and 

(g) generating a series of reports based on the answers, wherein different types of 
1 5 reports can be generated by selecting different groups as a basis of comparison. 

2. A method according to claim 1 in which the computer system includes a 
relational database, and at least some of the evaluation respondents are divided into more than 
one group, with the relational database being set up to enable the different groups to be selected 
as a basis of comparison. 

20 3. A method according to claim 2 in which the groups include primary groups and 

non-primary or secondary groups, with the groups being selected such that a respondent may 
belong to only one primary group and to one or more secondary groups. 

4. A method according to any one of the preceding claims in which the evaluation 
session comprises a number of substantially independent surveys each having a series of 

25 questions for answer by a respondent 

5. A method according to any one of the preceding claims which is operated in an 
internet browser/internet server environment. 
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6. A method according to claim 5 wherein the environment includes a host server 
having an associated relational database, at least one facilitator browser computer and a 
plurality of respondent computers, with the facilitator being involved in the initial selecting and 
creating steps. 

5 7. A method according to claim 4 wherein the series of questions are arranged to 

require a response whereby respondents rank subjects or objects, for enabling the generation of 
ranking reports. 

8. A method according to claim 5 wherein step (e) includes the steps of launching 
the evaluation process by including a universal resource locator (URL) of the address of a 

10 corresponding evaluation or survey to be carried out by the respondent. 

9. A computer system for producing an interactive evaluation comprising: 

respondent and group organisation means for inputting respondent and group details into 
said computer system of the respondents and groups participating in the evaluation; 

evaluation creation means for creating, for each respondent in a group of respondents, an 
15 interactive evaluation including a series of questions; 

notification means for notifying the respondents of said group of the availability of the 
corresponding interactive evaluations; 

evaluation execution means for enabling respondents to interact with a corresponding 
evaluation; 

20 storage means for storing the evaluation result in respect of each respondent, and 

reporting means for collating and reporting said evaluation results. 

10. A system as claimed in claim 9 further comprising: 

monitoring means for monitoring the current status of the interactive evaluation. 

11. A system as claimed in claim 10 wherein said monitoring means includes a 
25 second notification means for sending a reminder to respondents who have not completed their 

evaluation. 
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12. A system as claimed in claim 10 wherein the storage means comprises a 
relational database, and is arranged further to store details of the respondents, the groups, and 
the evaluations. 

13. A system as claimed in any one of claims 10 to 13 which includes registration 
5 means for initially registering an organisation's details with said computer system. 

14. A computer system according to any one of claim 9 to 13 wherein the respondent 
and group organisation means comprises a set up graphical user interface (GUT) for enabling a 
facilitator to enter and arrange names and into primary and secondary groups. 

15. A computer system according to any one of claims 9 to 14 wherein the evaluation 
10 creating means comprises at least one evaluation GUI including a survey* template option 

providing different survey types to choose from, and a question bank including questions 
associated with at least some of the survey types. 

16. A computer system according to claim 15 in which the survey type is chosen 
from a group including a self review, a colleague review, an opinion poll, and confidential and 

1 5 non-confidential reviews. 

17. A computer system according to any one of claim 9 to 16 in which the evaluation 
execution means includes a plurality of GUI's carrying respondent-specific survey questions. 

18. A computer system according to any one of the preceding claims 9 to 17 which 
includes closure means for enabling a facilitator to close an evaluation by force closure or 

20 deletion. 

19. A computer system according to any one of claims 9 to 18 wherein the reporting 
means includes select means for selecting a reports option available to an identified subject, the 
reports option being respondent-and group-based. 

20. A computer system according to claim 19 wherein the reports option includes a 
25 ranking option, whereby the identified subject can select which subject-accessible groups or 

respondents can be ranked by which other groups or respondents. 
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21. A computer system according to either one of claims 19 or 20 wherein the 
reporting means is arranged to process textual answers, and includes an edit facility to enable a 
user to moderate or bowdlerise the textual answers prior to saving them. 

22. A method of providing an evaluation on a computer system, the method 
5 comprising the steps of: 

providing an interface for facilitating the initiation of an interactive evaluation 

session; 

providing a respondent and group entry interface for enabling the entry of groups 
of evaluation respondents; 

10 providing an evaluation creation interface for enabling the compilation of at least 

one targeted survey for the respondents; 

providing an evaluation launching interface for launching the survey to the 

respondents; 

providing each of the respondents with the opportunity of interactively answering 
1 5 questions in the survey; 

receiving and storing on the computer system details of the evaluation session, 
respondents, groups, surveys and answers; and 

providing a report selection interface for enabling the selection of a series of 
reports on the survey. 

20 23. A method according to claim 22 wherein the group and respondent details on the 

evaluation session are stored on a relational database in such a way that allows for the selection 
of different combinations of groups at the reporting stage. 

24. A method according to either one of the preceding claims 22 or 23 which 
includes the step of allowing the entry of different groups of evaluation respondents as primary 
25 groups and secondary groups, with respondents being limited to membership of only one 
primary group, but to more than one secondary group. 
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25. A method according to claim 24 wherein the group and respondent details are 
stored in tables which include a group table containing details of the groups being evaluated, 
and a membership table including details of the primary group membership in respect of a given 
evaluation, and all secondary groups in respect of the evaluation. 

5 26. A computer system for enabling a computer-based evaluation, the system 

comprising: 

initiation means for generating an interface for facilitating the initiation of the 
interactive evaluation session; 

respondent and group entry interface generation means for enabling the entry of 
10 groups of evaluation respondents; 

evaluation creation interface generation means for enabling the compilation of at 
least one targeted survey for the respondents; 

evaluation launching generation interface means for launching the survey to the 

respondents; 

15 enabling means for providing each of the respondents with the opportunity of 

interactively answering questions in the survey; 

storage means for storing on the computer system details of the evaluation 
session, respondents, groups, surveys and answers, and 

report selection interface means for enabling the selection of a series of reports 

20 on the survey. 

27. A computer system according to claim 26 which includes server means 
incorporating storage means in the form of a relational database. 

28. A computer program product comprising a computer readable medium having its 
own program code means when said program is loaded, to make the computer execute 

25 procedures in accordance with the method of claims 22 to 25. 
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29. A computer system according to any one of claims 14 to 17 wherein the 
graphical user interfaces are in the form of web pages, and in combination define a web page 
database. 

30. A server hosting a web page database as claimed in claim 29. 



Dated this 3* day of June 2002 



Evalu8 Pty Limited 
by its attorneys 
Freehills Carter Smith Beadle 
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